Papers with MT evaluation metrics
Sampling and Filtering of Neural Machine Translation Distillation Data (2021.naacl-srw)
Copied to clipboard
| Challenge: | In most of neural machine translation distillation or stealing scenarios, the highest-scoring hypothesis of the target model is used to train a new model. |
| Approach: | They propose to use the highest-scoring hypothesis of the target model (teacher) to train a new model (student). |
| Outcome: | The proposed method improves the performance of MT models in English to Czech and with reference translations. |
Machine translation impact in E-commerce multilingual search (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that cross-lingual information retrieval performance is highly dependent on query translation quality. |
| Approach: | They investigate whether improving query translation quality yields little or no benefit to further improve retrieval performance. |
| Outcome: | The proposed methods compare query translations for multiple language pairs and identify the most promising language pairs to invest and improve. |
PyMarian: Fast Neural Machine Translation and Evaluation in Python (2024.emnlp-demo)
Copied to clipboard
| Challenge: | a Python interface to Marian NMT is available in PyPI via pip install pymarian . the interface provides a speedup factor of up to 7.8 the existing implementations . |
| Approach: | They propose a Python interface to Marian NMT, a C++-based training and inference toolkit for sequence-to-sequence models. |
| Outcome: | The proposed interface enables models trained with Marian to be connected to Python tools with a speedup factor of up to 7.8 the existing implementations. |
AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages (2024.naacl-long)
Copied to clipboard
Jiayi Wang, David Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokayed, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Mohamed, Hassan Ayinde, Oluwabusayo Awoyomi, Lama Alkhaled, Sana Al-azzawi, Naome Etori, Millicent Ochieng, Clemencia Siro, Njoroge Kiragu, Eric Muchiri, Wangari Kimotho, Toadoum Sari Sakayo, Lyse Naomi Wamba, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Iro, Saheed Abdullahi, Stephen Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Ogbu, Sam Ochieng’, Verrah Otiende, Chinedu Mbonu, Yao Lu, Pontus Stenetorp
| Challenge: | Recent advances in machine translation (MT) have focused on scaling multilingual machine translation models and evaluation data to hundreds of languages, including multiple under-resourced languages. |
| Approach: | They propose to use n-gram matching metrics to measure progress in multilingual machine translation to 13 typologically diverse African languages to create high-quality human evaluation data with simplified MQM guidelines. |
| Outcome: | The proposed metrics have a higher correlation with human judgments than n-gram matching metrics such as BLEU and METEOR. |
MTLens: Machine Translation Output Debugging (2022.lrec-1)
Copied to clipboard
Shreyas Sharma, Kareem Darwish, Lucas Pavanelli, Thiago Castro Ferreira, Mohamed Al-Badrashiny, Kamer Ali Yuksel, Hassan Sawaf
| Challenge: | a demo demonstrates a system for quantitatively evaluating MT systems in isolation or multiple MT models collectively . performance of machine translation systems varies significantly with inputs of diverging features, such as genres, genres and surface properties. |
| Approach: | They propose a benchmarking interface that quantitatively evaluates MT systems in isolation or collectively . the interface can be extended to include additional filters such as lexical, morphological, and syntactic features. |
| Outcome: | The proposed system quantitatively evaluates MT systems on multiple domains and evaluation metrics. |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |